LANGUAGE GENERATION TASKS

Adaptive Natural Language Generation for Task-oriented Dialogue via Reinforcement Learning (2022.coling-1)

Challenge:	Existing evaluation metrics focus on content selection, not linguistic quality . proposed GRUEN measures Grammaticality, non-redundancy, focUs, structure and coherence of generated text.
Approach:	They propose to use a BERT-based model and a class of syntactic, semantic, and contextual features to examine the system output.
Outcome:	Experiments show that the proposed metric correlates highly with human judgments.

Copied to clipboard

Atsumoto Ohashi, Ryuichiro Higashinaka

Retrieval Based Response Letter Generation For a Customer Care Setting (2022.naacl-industry)

Challenge:	In task-oriented dialogue systems, the role of the natural language generation component is to convert a system's intentions, called dialogue acts (DAs), into natural language utterances and to convey DAs accurately to users.
Approach:	They propose a method for Adaptive Natural language generation for Task-Oriented dialogue via Reinforcement learning that incorporates a natural language understanding module into the objective function of RL.
Outcome:	The proposed method generates adaptive utterances against speech recognition errors and the different vocabulary levels of users in a multi-world task-oriented dialogue system.

Copied to clipboard

Biplob Biswas, Renhao Cui, Rajiv Ramnath

Information Retrieval

Challenge:	Letter-like communications are a major means of customer relationship management . despite advances in natural language processing, the task of generating a response is time-consuming .
Approach:	They propose a deep-learning based response letter generation framework that uses data augmentation to retrieve knowledge from historical responses and utilize it to generate an appropriate response.
Outcome:	The proposed model outperforms baselines by significant margins while producing consistent and informative responses.

DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantization (2022.acl-short)

Copied to clipboard

Zheng Li, Zijian Wang, Ming Tan, Ramesh Nallapati, Parminder Bhatia, Andrew Arnold, Bing Xiang, Dan Roth

GLGE: A New General Language Generation Evaluation Benchmark (2021.findings-acl)

Challenge:	Empirical analyses show that pre-trained sequence-to-sequence models can achieve a 16.5x model footprint compression ratio with little performance drop relative to full-precision counterparts.
Approach:	They propose to distill and quantize pre-trained sequence-to-sequence models to reduce memory and latency requirements.
Outcome:	Empirical results show that the proposed model achieves 16.5x model footprint compression ratio with little performance drop relative to full-precision counterparts on multiple summarization and QA datasets.

Copied to clipboard

Dayiheng Liu, Yu Yan, Yeyun Gong, Weizhen Qi, Hang Zhang, Jian Jiao, Weizhu Chen, Jie Fu, Linjun Shou, Ming Gong, Pengcheng Wang, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, Ruofei Zhang, Winnie Wu, Ming Zhou, Nan Duan

PRAL: A Tailored Pre-Training Model for Task-Oriented Dialog Generation (2021.acl-short)

Challenge:	Multi-task benchmarks focus on a range of Natural Language Understanding (NLU) tasks without considering the Natural Language Generation (NLG) models.
Approach:	They propose a multi-task benchmark for evaluating the generalization capabilities of NLG models across eight language generation tasks.
Outcome:	The proposed benchmarks are based on GLUE and Su-perGLUE for English and several other languages.

Copied to clipboard

Jing Gu, Qingyang Wu, Chongruo Wu, Weiyan Shi, Zhou Yu

Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning (2020.findings-emnlp)

Challenge:	Existing approaches to building task-oriented dialog systems require a substantial amount of annotations and thus are labor-intensive.
Approach:	They propose a Pre-trainedRole Alternating Language model (PRAL) that is explicitly designed for task-oriented dialog tasks.
Outcome:	The proposed model outperforms or is on par with state-of-the-art models on task-oriented dialog tasks.

Copied to clipboard

Zhaojiang Lin, Andrea Madotto, Pascale Fung

Revisiting Representation Degeneration Problem in Language Modeling (2020.findings-emnlp)

Challenge:	Large-scale language models can be fine-tuned to learn highly transferable embedding, but they are expensive and require multiple model parameters.
Approach:	They propose a way to fine-tune multiple down-stream generation tasks simultaneously using a single, large pretrained model.
Outcome:	The proposed model can maintain or improve the performance of fine-tuning the whole model.

Copied to clipboard

Zhong Zhang, Chongming Gao, Cong Xu, Rui Miao, Qinli Yang, Junming Shao

On Decoding Strategies for Neural Text Generators (2022.tacl-1)

Challenge:	Language modeling is a fundamental task in natural language processing, applications include machine translation, image captioning and speech recognition.
Approach:	They propose a cosine regularization method to solve the representation degeneration problem by analyzing the limitations of the proposed method and then propose an alternative regularization technique to tackle the problem.
Outcome:	The proposed method is effective in language modeling and image captioning.

Copied to clipboard

Gian Wiher, Clara Meister, Ryan Cotterell

Rethinking and Refining the Distinct Metric (2022.acl-short)

Challenge:	a recent study suggests that decoding strategies may be more important than the model architecture itself when generating text from probabilistic models.
Approach:	They propose to measure changes in attributes of generated text as a function of decoding strategy and task using human and automatic evaluation.
Outcome:	The proposed study shows that decoding strategies do not always transfer across tasks . authors show that the differences in attributes are not always consistent across tasks, they say .

Copied to clipboard

Siyang Liu, Sahand Sabour, Yinhe Zheng, Pei Ke, Xiaoyan Zhu, Minlie Huang

Challenge:	Existing methods for calculating distinct scores have evident biases that assign higher penalties to longer sequences.
Approach:	They propose to scale the number of distinct tokens based on their expectations.
Outcome:	The proposed metric removes evident biases in the original distinct score . the proposed meter correlates better with human judgment in evaluating response diversity .

M-RAG: Reinforcing Large Language Model Performance through Retrieval-Augmented Generation with Multiple Partitions (2024.acl-long)

Copied to clipboard

Zheng Wang, Shu Teo, Jieer Ouyang, Yongjun Xu, Wei Shi

Language Generation via Combinatorial Constraint Satisfaction: A Tree Search Enhanced Monte-Carlo Approach (2020.findings-emnlp)

Challenge:	Existing methods for retrieving relevant memories from an external database are coarse-grained and can cause noise and focus on crucial memories.
Approach:	They propose a multiple partition paradigm for RAG where each database partition serves as a basic unit for execution.
Outcome:	The proposed framework outperforms baseline methods on three language generation tasks on seven datasets.

Copied to clipboard

Maosen Zhang, Nan Jiang, Lei Li, Yexiang Xue

Searching for Search Errors in Neural Morphological Inflection (2021.eacl-main)

Challenge:	Generating natural language under complex constraints is a principled formulation towards controllable text generation.
Approach:	They propose a method to specify combinatorial constraints for sentence generation . they use a tree search algorithm embedded into the proposal process of the Markov Chain Monte Carlo .
Outcome:	The proposed method achieves consistent and significant improvement on multiple language generation tasks.

Copied to clipboard

Martina Forster, Clara Meister, Ryan Cotterell

Syntactic Text Processing

Challenge:	Neural sequence-to-sequence models are the predominant choice for language generation tasks.
Approach:	They find that on word-level tasks, the empty string is often the global optimum . they suggest that the poor calibration of many neural models may stem from characteristics of a specific subset of tasks rather than general ill-suitedness of such models for language generation.
Outcome:	The results suggest that the poor calibration of many neural models may stem from characteristics of a specific subset of tasks rather than general ill-suitedness of such models for language generation.

A Reinforcement Learning Framework for Natural Question Generation using Bi-discriminators (C18-1)

Copied to clipboard

Zhihao Fan, Zhongyu Wei, Siyuan Wang, Yang Liu, Xuanjing Huang

If beam search is the answer, what was the question? (2020.emnlp-main)

Challenge:	Existing research on visual question generation is focused on training models to fit the annotated data set that makes them indifferent from other language generation tasks.
Approach:	They propose to use two discriminators to enhance the training of a visual question generator to ask natural questions about an image.
Outcome:	The proposed model outperforms state-of-the-art models in terms of automatic and human evaluation metrics.

Copied to clipboard

Clara Meister, Ryan Cotterell, Tim Vieira

Hierarchical Recurrent Aggregative Generation for Few-Shot NLG (2022.findings-acl)

Challenge:	surprisingly, beam search results on language generation tasks are low-quality . despite its high error rate, beam searches can be used to decode models with high probability .
Approach:	They frame beam search as the exact solution to a different decoding objective . they propose a set of decoding objectives that explicitly enforce this property .
Outcome:	The proposed method enforces uniform information density in text, a property motivated by cognitive science.

Copied to clipboard

Giulio Zhou, Gerasimos Lampouras, Ignacio Iacobacci

HistAlign: Improving Context Dependency in Language Generation by Aligning with History (2023.emnlp-main)

Challenge:	Existing approaches do not account for the fact that some sub-tasks, specifically aggregation and lexicalisation, can benefit from transfer learning in different extents.
Approach:	They propose a hierarchical approach for few-shot and zero-shot generation using a three-moduled jointly trained architecture.
Outcome:	The proposed approach achieves state-of-the-art on few-shot and zero-shot settings compared to previous approaches.

Copied to clipboard

David Wan, Shiyue Zhang, Mohit Bansal

Informed Sampling for Diversity in Concept-to-Text NLG (2021.findings-emnlp)

Challenge:	Language models (LMs) can generate hallucinations and incoherent outputs due to their weak context dependency.
Approach:	They propose a training approach to ensure good cache alignment so that the model receives useful signals from the history.
Outcome:	The proposed approach improves text coherence and faithfulness on diverse language generation tasks.

Copied to clipboard

Giulio Zhou, Gerasimos Lampouras

Instruct Once, Chat Consistently in Multiple Rounds: An Efficient Tuning Framework for Dialogue (2024.acl-long)

Challenge:	Existing methods to encourage lexical diversity for language generation tasks produce repetitive outputs, but this often comes at a cost to the perceived fluency and adequacy of the output.
Approach:	They propose to augment the decoding process with a meta-classifier trained to distinguish which words at any given timestep will lead to high-quality output.
Outcome:	The proposed method achieves a high level of diversity with minimal effect on the output’s fluency and adequacy.

Copied to clipboard

Jian Wang, Chak Tou Leong, Jiashuo Wang, Dongding Lin, Wenjie Li, Xiaoyong Wei

Enhancing Language Generation with Effective Checkpoints of Pre-trained Language Model (2021.findings-acl)

Challenge:	Tuning language models for dialogue generation has been a prevalent paradigm for building capable dialogue agents.
Approach:	They propose a multi-round interactive dialogue tuning framework that models the speaker roles of agent and user separately.
Outcome:	The proposed framework performs superior to fine-tuning and improves dialogue consistency.

Copied to clipboard

Jeonghyeok Park, Hai Zhao

DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation (2023.eacl-main)

Challenge:	Existing methods to exploit PrLMs for NLG tasks do not get as much performance gain as in the NLU task.
Approach:	They propose a method to integrate public checkpoints of PrLMs for the most convenience.
Outcome:	The proposed method significantly improves the quality of the language generation tasks on 6 different kinds of PrLMs.

Copied to clipboard

Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, Ali Ghodsi

Toward Human-Like Evaluation for Natural Language Generation with Error Analysis (2023.acl-long)

Challenge:	Pre-training/fine-tuning of pre-training models has become more expensive and resource-hungry.
Approach:	They propose a low-rank adaptation technique that trains LoRA blocks for a range of ranks instead of a single rank.
Outcome:	The proposed method trains LoRA blocks for a range of ranks instead of a single rank . it can train dynamic search-free models with DyLoRA at least 4 to 7 times faster than LoRA .

Copied to clipboard

Qingyu Lu, Liang Ding, Liping Xie, Kanjian Zhang, Derek F. Wong, Dacheng Tao

He is very intelligent, she is very beautiful? On Mitigating Social Biases in Language Modelling and Generation (2021.findings-acl)

Challenge:	Pre-trained language models (PLMs) have been used to evaluate language generation tasks . pretrained error analysis can be used to refine the generated sentence toward higher confidence .
Approach:	They propose to combine pretrained language model based metrics with human-like error analysis to improve sentence confidence.
Outcome:	The proposed method outperforms top-scoring metrics in 19/25 settings.

Copied to clipboard

Aparna Garimella, Akhash Amarnath, Kiran Kumar, Akash Pramod Yalla, Anandhavelu N, Niyati Chhaya, Balaji Vasan Srinivasan

Fine-tuned Language Models are Continual Learners (2022.emnlp-main)

Challenge:	Existing studies have focused on mitigating social biases in context-free representations, with recent shift to contextual ones.
Approach:	They propose an approach to mitigate social biases in a large pre-trained contextual language model . they propose lexical co-occurrence-based bias penalization in the decoder units .
Outcome:	The proposed approach reduces biases in fill-in-the-blank sentences and summarizes . it also reduces the biased representations in the frameworks, the authors show .

Copied to clipboard

Thomas Scialom, Tuhin Chakrabarty, Smaranda Muresan

Pragmatically Informative Text Generation (N19-1)

Challenge:	Recent work on large language models relies on intuition that most tasks can be described via natural language instructions.
Approach:	They propose that a model should be able to keep extending its knowledge without forgetting previous skills.
Outcome:	The proposed model can learn 8 new diverse language generation tasks while maintaining good performance on previous tasks, spanning in total of 70 datasets.

Copied to clipboard

Sheng Shen, Daniel Fried, Jacob Andreas, Dan Klein

Factual or Contextual? Disentangling Error Types in Entity Description Generation (2023.acl-long)

Challenge:	Existing approaches to pragmatics have been used to improve the informativeness of generated text in grounded language learning problems.
Approach:	They propose to use pragmatics to improve the informativeness of conditional text models . they propose to apply pragmatic reasoning to more traditional language generation tasks .
Outcome:	The proposed methods improve the performance of strong existing systems for abstractive summarization and generation from structured meaning representations.

Copied to clipboard

Navita Goyal, Ani Nenkova, Hal Daumé III

BERTGen: Multi-task Generation through BERT (2021.acl-long)

Challenge:	Existing evaluation practices only distinguish between model generated referring expressions being accurate (ground-truth) versus inaccurate (not groundtruth).
Approach:	They propose to integrate indicators for factual inconsistencies and contextual incongruities into automated evaluations of language models to assess the differences in error types across familiar vs unfamiliar entities.
Outcome:	The proposed evaluation paradigm disentangles factuality and congruity errors in natural contexts.

Copied to clipboard

Faidon Mitzalis, Ozan Caglayan, Pranava Madhyastha, Lucia Specia

Gender Biases in Automatic Evaluation Metrics for Image Captioning (2023.emnlp-main)

Challenge:	Recent work in unsupervised and self-supervised pre-training has revolutionised the field of natural language understanding (NLU).
Approach:	They propose to use multimodal and multilingual pre-trained models to extend BERT by fusing them together for language generation tasks.
Outcome:	The proposed model outperforms baseline models in image captioning, machine translation and multimodal machine translation tasks and is competitive with supervised counterparts.

Copied to clipboard

Haoyi Qiu, Zi-Yi Dou, Tianlu Wang, Asli Celikyilmaz, Nanyun Peng

Multimodality

Challenge:	Pretrained evaluation metrics can perpetuate and amplify biases, causing inability to differentiate between biased and unbiased generations.
Approach:	They conduct a systematic study of gender biases in image captioning tasks . they show that pretrained models perpetuate and amplify biase .
Outcome:	The proposed model-based evaluation metrics have shown good correlations with human judgments in language generation tasks.

NEUROSTRUCTURAL DECODING: Neural Text Generation with Structural Constraints (2023.acl-long)

Copied to clipboard

Mohaddeseh Bastan, Mihai Surdeanu, Niranjan Balasubramanian

Syntactic Text Processing

Challenge:	Current approaches for conditional text generation focus on lexical constraints, but lack syntactic constraints to support complex semantic constraints.
Approach:	They propose a decoding algorithm that incorporates syntactic constraints to improve the quality of the generated text.
Outcome:	The proposed method improves on three different language generation tasks and shows improved lexical and syntactic metrics.

Hit the Sweet Spot! Span-Level Ensemble for Large Language Models (2025.coling-main)

Copied to clipboard

Yangyifan Xu, Jianghao Chen, Junhong Wu, Jiajun Zhang

Robustness in NLP

Challenge:	a recent study focused on sample-level and token-level ensembles, which hinder dynamic correction and enhancement of outputs during the generation process.
Approach:	They propose a span-level ensemble method that balances real-time adjustments and accurate ensemble decisions.
Outcome:	The proposed method improves performance across language generation tasks significantly.

Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach (2021.emnlp-main)

Copied to clipboard

Haoming Jiang, Bo Dai, Mengjiao Yang, Tuo Zhao, Wei Wei

Generalized Entropy Regularization or: There’s Nothing Special about Label Smoothing (2020.acl-main)

Challenge:	Existing methods for evaluation of dialog systems are expensive and not scalable . a framework for estimating human evaluation scores is proposed to bridge this gap .
Approach:	They propose a framework for estimating human evaluation scores based on off-policy evaluation . they use language quality metrics for single-turn response generation given a fixed context .
Outcome:	The proposed framework outperforms existing methods in terms of correlation with human evaluation scores.

Copied to clipboard

Clara Meister, Elizabeth Salesky, Ryan Cotterell

DORB: Dynamically Optimizing Multiple Rewards with Bandits (2020.emnlp-main)

Challenge:	Prior work has explored regularizing the output distributions of probabilistic models to alleviate overfitting.
Approach:	They propose a family of entropy regularizers that have a connection to regularization . they find that label smoothing provably does not allow for sparsity in an output distribution .
Outcome:	The proposed method improves the relationship between model entropy and performance on language generation tasks.

Copied to clipboard

Ramakanth Pasunuru, Han Guo, Mohit Bansal

PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation (2020.emnlp-main)

Challenge:	Recent advances in end-to-end neural networks-based approaches have shown wide success in sequence generation tasks.
Approach:	They propose to optimize multiple metric rewards simultaneously using a multi-armed bandit approach . they empirically show the effectiveness of their approaches via various automatic metrics and human evaluation .
Outcome:	The proposed approach improves on question generation and data-to-text generation using a bandit approach.

Copied to clipboard

Bin Bi, Chenliang Li, Chen Wu, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, Luo Si

Natural Language Interfaces

Challenge:	Existing techniques for natural language understanding and generation use autoencoding and/or autoregressive objectives to train models.
Approach:	They propose a self-supervised pre-training scheme that pre-trains an autoencoding and autoregressive language model on a large unlabeled corpus for generating new text conditioned on context.
Outcome:	The proposed scheme achieves state-of-the-art results on a variety of language generation benchmarks covering generative question answering, abstractive summarization and conversational response generation.

Prediction-Augmented Generation for Automatic Diagnosis Tasks (2025.findings-acl)

Copied to clipboard

Chan-Yang Ju, Dong-Ho Lee